Quality reporting for assessment data analysis platform

ABSTRACT

A platform is described for analyzing assessment data, including correlating assessment data with learning outcome data, and presenting analysis results through one or more interactive dashboards. A course syllabus and/or other course described information is analyzed to identify expected target outcomes to be achieved by students of the course, and to identify a relative weight of each of the various target outcomes based on their coverage in the syllabus. Assessment data describing a digitally administered assessment is analyzed to identify the categories that are assessed in the various questions of the test, and determine student performance with respect to particular categories. The platform provides interactive dashboards to present analysis results and provide integration with a continuous quality improvement (CQI) system for task tracking related to the analysis results.

CROSS-REFERENCE TO RELATED APPLICATION

The present disclosure is related to, and claims priority to, U.S.Provisional Patent Application Ser. No. 62/520,366, titled “OutcomeCorrelation Analysis for Testing,” which was filed on Jun. 15, 2017, andthe entirety of which is incorporated by reference into the presentdisclosure.

BACKGROUND

Institutions and organizations, within an academic, educational, orother setting, which have strategic plans with key goals and which seekto continuously improve processes, are adopting a variety of softwareproducts in order to collect data in an electronic format. However,despite the adoption of these software products, the collection,integration, and analysis of the data to continuously improve processesis sometimes done manually, which is typically labor intensive anderror-prone. Further, these previously available software products maynot provide readily usable information or insights, which supportevidence-based decision making, in a comprehensive, consistent, andeasily understood format for the achievement of strategic plans and keygoals. Moreover, a variety of software products are currently availableto administer tests and/or other types of assessments for education,professional certification, licensing certification, and other purposes.Such products may be designed to address specific needs within anacademic, educational, or other setting, and accordingly may not providereadily usable information for analysis or assessment, in a consistentand easily understood format.

SUMMARY

Implementations of the present disclosure are generally directed tomechanisms to create, measure, monitor, and/or evaluate key goals fromstrategic plans in educational and/or other environments. Moreparticularly, implementations of the present disclosure are directed tothe integration and analysis of data from diverse sources within aprogram, comparing the designed and intended curriculum mapping ofeducational outcomes to the taught, assessed, and/or learned curriculummapping of educational outcomes in a course or set of courses,identifying gaps, redundancies and inefficiencies, and presenting thisinformation and/or other information in one or more dynamicallyinteractive graphic user interfaces (UIs), also described as dashboards.Implementations of the present disclosure are also directed tointegrating the dashboard(s) with a system for creating and managingtickets including continuous quality improvement (CQI) tasks andobservations to enable users of the dashboard(s) to generate ticketsthat are assigned to responsible individuals and tracked to monitor theprogress of the responsible individuals in completing the tasks. The CQItasks document and track data-based decisions and how they advancequality improvement efforts. Implementations of the present disclosureare also directed to integrating and analyzing the data from diverseacademic assessment sources, and comparing the results of the analysisto various types of (e.g., external peer and national) data, todetermine strengths, weaknesses, opportunities, and threats related totheir program effectiveness, competitive programs, national standards,and academic accreditation standards of the various types of analyzeddata. Such strengths, weaknesses, opportunities, and threats analysisand other implementations are described in more detail below.

In general, implementations of innovative aspects of the subject matterdescribed in this specification can be embodied in a method thatincludes the following operations: receiving, by a platform, assessmentdata from an assessment service that digitally administers at least oneassessment, the assessment data including category data describing atleast one category associated with each respective question of the atleast one assessment, the assessment data further including score datadescribing a plurality of scores each assigned based on a response of anindividual to a question on the at least one assessment; analyzing, bythe platform, the assessment data to determine analysis results,including aggregating the score data for each respective question of theat least one assessment; presenting the analysis results through atleast one interactive dashboard provided by the platform; and receiving,through the at least one interactive dashboard provided by the platform,a request to create a ticket in a continuous quality improvement (CQI)system that is accessible through the platform and, in response to therequest, creating the ticket in the CQI system, wherein the ticket iscreated to include a screenshot of a current presentation of the atleast one interactive dashboard when the ticket is created.

These and other implementations can each optionally include one or moreof the following innovative aspects: the operations further includenormalizing, by the platform, the assessment data to provide a samerange of scores for a plurality of questions described in the assessmentdata; analyzing the assessment data further includes determining, foreach respective question, a point biserial and a percentage correctmetric that indicates a percentage of students who answered the questioncorrectly, based on the score data for the respective question, anddetermining a quality of each question based on its point biserial andpercentage correct metric; the request to create the ticket indicates atleast one question for which the determined quality is low, as indicatedby the point biserial and percentage correct metric each being below arespective threshold value; the operations further include receiving,through the at least one interactive dashboard provided by the platform,an indication that the ticket is to be resolved and, in responseresolving the ticket, and updating the ticket to include a screenshot ofa current presentation of the at least one interactive dashboard whenthe ticket is resolved; the operations further include presenting,through that at least one interaction dashboard, change information thatdescribes a different in at least a portion of the assessment data fromwhen the ticket is resolved relative to when the ticket was created; therequest to create the ticket indicates one or more of a question, anassessment that includes the question, a course in which the assessmentis administered, and an individual associated with the course; therequest to create the ticket indicates at least one action to beperformed, at least one responsible individual to perform the at leastone action; a notification is sent to the at least one responsibleindividual responsive to creating the ticket in the CQI system; and/orthe operations further include generating, through the at least oneinteractive dashboard, a report that includes a list of tickets in theCQI system that have been created based on requests submitted throughthe at least one interactive dashboard.

Other implementations of any of the above aspects include correspondingsystems, apparatus, and/or computer programs that are configured toperform the actions of the method(s). The present disclosure alsoprovides a computer-readable storage medium coupled to one or moreprocessors and having instructions stored thereon which, when executedby the one or more processors, cause the one or more processors toperform operations in accordance with implementations of the methodsprovided herein. The present disclosure further provides a system forimplementing the methods provided herein. The system includes one ormore processors, and a computer-readable storage medium coupled to theone or more processors having instructions stored thereon which, whenexecuted by the one or more processors, cause the one or more processorsto perform operations in accordance with implementations of the methodsprovided herein.

It is appreciated that implementations in accordance with the presentdisclosure can include any combination of the aspects and featuresdescribed herein. That is, implementations in accordance with thepresent disclosure are not limited to the combinations of aspects andfeatures specifically described herein, but also include any otherappropriate combinations of the aspects and features provided.

The details of one or more implementations of the present disclosure areset forth in the accompanying drawings and the description below. Otherfeatures and advantages of the present disclosure will be apparent fromthe description and drawings, and from the claims.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 depicts an example platform for assessment data analysis,according to implementations of the present disclosure.

FIG. 2A depicts an example of outcome data that may be employed inoutcome correlation analysis, according to implementations of thepresent disclosure.

FIG. 2B depicts an example of category data that may be employed inoutcome correlation analysis, according to implementations of thepresent disclosure.

FIG. 2C depicts a flow diagram of an example process for normalization,according to implementations of the present disclosure.

FIG. 3 depicts a flow diagram of an example process for assessment dataanalysis, according to implementations of the present disclosure.

FIGS. 4-10 depict examples of dashboards that present assessmentinformation, correlation analysis results, and/or other information,according to implementations of the present disclosure.

FIG. 11 depicts an example computing system, according toimplementations of the present disclosure.

DETAILED DESCRIPTION

Implementations of the present disclosure are directed to a platform forextracting, collecting, integrating, analyzing, and presentingassessment data from an academic environment and/or other types ofenvironments. Implementations also provide for various interactivedashboards that provide data visualization for the results of theanalysis, integrated with a system for creating and managing ticketscomprising of CQI tasks and observations, and provide an automatedprocess to monitor and evaluate academic assessment data againstexternal criteria promulgated by an accreditation organization or othertype of entity. Implementations also provide an automated process toevaluate institutions or organizations against external criteriapromulgated by an accreditation organization or other type of entity.

Academic institutions such as universities, colleges, professionalschools, trade schools, and/or other types of organizations performvarious types of assessments of their students. Such assessmentsgenerate a large amount of data that, using previously availablesolutions, was stored in various disparate sources with differingformats. Moreover, institutions have traditionally lacked the processand tools needed to integrate, analyze and understand the data, identifystrengths, weaknesses, opportunities, and threats related to theirprogram effectiveness, competitive programs, national standards, andacademic accreditation standards, identify action items to addressweaknesses, opportunities and threats in the data or to documentstrengths, recognize trends exhibited by the data, and so forth. Theimplementations described herein provide a platform to address theshortcomings in previously available solutions and, in some cases,provide a solution for extraction, collecting, integration, and analysisof assessment data that was previously unavailable. The platform canintegrate, analyze, and assist users in understanding the data, andidentify areas of improvement.

As used herein, an assessment describes any process by which anindividual understanding and/or knowledge (e.g., student) is evaluatedaccording to one or more objective or subjective metrics (e.g., programlearning goals). An assessment can be one or more of the following: asurvey, an assignment, a quiz, a test, an observation, a rubric, avocational assessment, a clinical evaluation, and/or other appropriatetype of assessment. An assessment can produce a result (e.g., a score orgrade) that provides a gauge of the assessed individual's proficiencywith respect to one or more academic outcome. An assessment can includeany number of items, also described herein as questions.

The platform described herein integrates and analyzes data from one ormore assessment services, administrated and/or recorded via a variety ofsoftware products, the data including score data for the assessment ofstudent achievement of learning outcomes, course objectives, programoutcomes and, national learning outcomes. In some examples, anassessment can be administered using software, and the score data can begenerated and stored through operations of the software, based on thestudents' input to the software during the assessment. In some examples,an assessment can be administered at least partly outside of a softwareenvironment, such as a practical assessment to determine the student'sparticular level of proficiency at a skill or craft such as welding,driving, public speaking, musical performance, and so forth. In suchexamples, the score data can be entered to the software by an instructoror other user. The grading of observations can be recorded in thesoftware through any suitable rubric. The platform provides interactivedashboards to present the results of the data integration and/oranalysis, including metrics such as key performance indicators (KPIs)that are relevant to the particular outcomes of an institution and/oreducator administrating the assessments. Such outcomes can be learningoutcomes, requirements and/or standards provided by an accreditationorganization, or other type of entity, that determines the institution'squality and/or whether the institution is to be accredited.

In some implementations, the assessment service(s) include digitalassessment services such as ExamSoft™. The platform can extract,integrate, and/or analyze data provided by any suitable assessmentservice. The data received from an assessment service can include thequestions asked on assessments, type of question (e.g., multiple choice,true/false, fill in the blank, essay, rubric etc.), possible answers toquestions, number of possible answers, and so forth. The data can alsoinclude psychometric data regarding the questions asked on assessments,such as difficulty index—the percent of students who answered thequestion correctly, point biserial correlation which measures thereliability of assessment questions, KR-20 which measures thereliability of inter-question consistency, how many students selectedeach possible answer, how many points each student got for each answer,and so forth. The data can also include category data describing thevarious categories that have been associated with each question, whichcan also be described as the labeling of a question. Category data isdescribed further below, and can include a learning outcome, learningobjective, topic or/and subject of the question (to any suitable degreeof specificity), a course and/or assessment associated with thequestion, and so forth. Questions may be associated with any appropriatenumber of categories. In some instances, a question may have multiplecategories with different levels of categorization. For example, aquestion can have one or more primary categories, secondary categories,tertiary categories, and so forth.

In some implementations, the analysis of the data may includenormalization of the data received from assessment services. Suchnormalization can include normalizing a scale of possible scores for thequestions, such as normalizing scores to a scale from 0 to 1 where 0 isno credit and 1 is full credit for a question. For each assessedindividual, also described herein as a student, a determination may bemade of the overall weighted percentage of correct questions on anassessment, as a sum of the scale for each question that is weightedaccording to the normalization. For example, a determination may be madethat a particular student earned, on a particular assessment, 82.5% ofthe credit available for the questions on the assessment. Suchnormalization makes it possible to compare the performance of studentson different assessments that may have different numbers of questionsand/or different question weight contribution to the total pointsavailable on the assessment. The normalization also enables comparing ofstudent performance between different students in the same or differentcourses, and comparison of student performance over time. Normalizationcan be performed across assessments and across questions in anassessment, such that a normalized scale of credit is applied toquestions that may be directed to different categories in one or moreassessments.

In some implementations, different questions on an assessment can beweighted differently based on the difficulty of the questions and/orother criteria. The weights may be taken into account when determining astudent's score on a particular assessment, or across multipleassessments administered in a course. For example, a student's score canbe a weighted total or weighted average of the scores on the variousquestions, weighted according to the weights assigned to the differentquestions. Such weighting may be determined based on historical dataregarding answers to questions, where such historical data is analyzedto determine the difficulty of a question. For example, hard questionsmay be weighted at 3 points, medium questions weighted at 2 points, andeasy questions weighted at 1 point. The weight may be applied as acoefficient when determining the overall score of a student on anassessment and/or in a course. In some implementations, thenormalization described herein may be performed instead of weightingdifferent questions. In some implementations, questions may benormalized as described herein, and weights may be further applied tothe normalized questions to provide different weights according tocategory or other considerations.

A question on an assessment can be any appropriate portion of anassessment that assesses the student with respect to one or moreparticular pieces of knowledge and/or skills. For example, a writtenassessment (e.g., digitally administered or otherwise) can include 20multiple choice questions regarding European history or some othertopic. As another example, an orally administered or practicalassessment may include 5 questions that are areas of assessment, such asassessing a medical student with respect to five different areas whereclinical proficiency is desired. A question is also described herein asan item, and may include any suitable instance in which a student isassessed with respect to a particular piece of knowledge or skill, withany suitable degree of specificity. For example, an item may be apractical determination of skill, such as a determination of a student'sability to weld or perform other physical activities outside of, or inaddition to, a written examination scenario.

Each question can be associated with any suitable number of categories.In some instances, categories can be categories from the Bloom'staxonomy or any other cognitive dimensions, as described further below.A category of a question can also indicate the assessment that includesthe question, the course in which the question was administered as partof an assessment, a professor that administered the course, assessment,and/or question, and so forth. A category can also indicate a desiredlearning outcome for the question, a topic or subject of the question,and so forth. The platform can analyze the data received from theassessment service(s) to identify questions that are not associated withcategories, and/or that are associated with too few or too manycategories. For example, a question may be associated with a categoryidentifying the course, but may lack additional categories indicating alearning outcome, topic, and/or subject of the question. Suchshortcomings of the received data may be flagged by the platform, andCQI tasks (described below) may be generated, with a direct web link tothe questions in the assessment system (e.g., ExamSoft™), to prompt aresponsible individual (e.g., the professor teaching the course) toprovide/label the missing categories for the flagged questions.

The received data can be integrated and analyzed according to variouscriteria. For example, data can be aggregated for a course, to includethe assessments administered in the course. Data can be integrated andanalyzed for a course for a particular year, or across multiple years.Data can be integrated and analyzed across multiple courses, such asacross all the courses administered as part of an educational program(e.g., pharmacy school courses administered in a cohort, the first yearor in all years of the program).

In some implementations, the received input data from assessmentservices may be modified as part of a pre-processing phase prior tosubsequent analysis. For example, data can be modified to normalizeand/or otherwise alter the categories of the questions described in thedata. In some instances, incoming category data can be described as ahierarchy of categories, arranged in a tree structure with a top-levelcategory, various subcategories under the top-level category, varioussub-subcategories under each subcategory, and so forth. Such a hierarchycan be flatted, such that each category is assigned a name thatdescribes its original hierarchical position (e.g., a full path of acategory within a hierarchy). Modification of category data can alsoinclude normalization to provide consistency in category name andidentification.

In some implementations, the platform can also receive courseinformation that describes the desired or targeted learning outcomes fora course. Such course information can be retrieved from a coursesyllabus or other course description. In some instances, courseinformation can be input to the platform by a professor or otherindividual associated with the course.

The course information, indicating learning outcomes for a course, canbe correlated with the score data and category data for the questionsincluded in the assessments administered as part of the course. Suchcorrelation can produce output data that indicates the extent to which acourse is actually administering assessments that evaluate students withrespect to the desired learning outcomes. For example, an introductorycalculus course may have learning outcomes indicating that students ofthe course are to learn how to evaluate integrals, evaluate derivatives,and solve simple differential equations. If the assessments administeredduring the course have questions associated with learning outcomecategories for integrals and derivatives, but lack (or have few)questions associated with differential equation learning outcomes, theparticular course may be flagged as having assessments that do notadequately correspond to the desired learning outcomes. Learningoutcomes can be an aggregate of multiple courses and learning outcomesover a period of semesters or years. So, while a learning outcome for aparticular course might be achieved, the aggregate learning outcomemight still fail in some instances. For example, an introductorycalculus course learning outcome might be achieved, but it could be partof an aggregate for a learning outcome of math needed for civilengineering learning outcome, and that learning outcome might fail eventhough the more specific learning outcome may have been achieved.

The correlation can also produce output data that indicates how wellstudents are performing with respect to the learning outcomes and/or anyother objective of a course, based on the scores the students earn onquestions that have categories associated with the learning outcomesand/or any other objective. For example, following the calculus courseexample above, if students in the course are achieving adequate scoreson questions that have categories for integrals and derivatives, but areachieving below-threshold scores on questions that have categories fordifferential equations, a determination may be made that the course isinadequately teaching students about differential equations, and theprofessor may be presented with an action item, a CQI tasks, to improvetheir performance in that particular learning outcome. Correlation mayalso take into account overall student performance, question qualitydeterminations, and/or other factors that are determined based on theanalysis, as described further below.

The platform provides various interactive dashboards as UIs that a usercan employ to view the results of the analysis and/or perform actionsassociated with the assessment data and analysis results. Suchdashboards are described as interactive and/or dynamic, given that thedashboards can alter their appearance dynamically, in real time, inresponse to actions performed by the user within the dashboard tospecify thresholds, define data filters, and/or otherwise requestalteration of the data visualization.

In some implementations, the dashboards provided by the platformincludes a course effectiveness dashboard. The course effectivenessdashboard may present a scatter plot showing psychometric data regardinga particular assessment (e.g., exam), and/or a scatter plot for all theassessments administered in a particular course. The course and/orassessment may be selected from drop-down menus and/or other suitable UIcontrols included on the dashboard. In some implementations, the scatterplot may present a series of points each corresponding to a questionadministered on an assessment. The points may be plotted on the scatterplot, with the X-axis indicating the percentage of students who answeredthe question correctly, and the Y-axis indicating the point biserial ofthe question. Further, the course effectiveness dashboard may presentthe percent of difficult, intermediate, and/or easy questions perassessment and the student performance on categories associated with thequestions. An example scatter plot is shown in FIG. 4.

In some implementations, the calculation of the point biserial for aquestion includes designating the two groups of students, including anupper group of students (e.g., the upper 27%) and a lower group ofstudents (e.g., the lower 27%). A degree of correlation is thendetermined between the two groups, such as a comparison of the number ofstudents in the upper group who answered the question correctly versusthe number of students in the lower group who answered the questioncorrectly. The point biserial for a question indicates the amount ofpositive, or negative correlation between the performance of the twogroups on the particular question. A quality of the question may beinferred based on its point biserial. For example, if the students whoanswered the question correctly are mostly the students who performedwell on the assessment in general (e.g., the upper group), and thestudents who answered the question incorrectly are mostly the studentswho did not perform well (e.g., the lower group), then the question canbe inferred as being a good question. If the question was answeredcorrectly by a similar or same proportion of both groups of students,the question may be designated a low-quality question, given that bothgroups of students were answering the question randomly. A low-qualityquestion may have a point biserial that is low (e.g., negative),compared to a high-quality question that has a high (e.g., positive)point biserial.

The scatter plot may use the two metrics, percentage of studentsanswering correctly (e.g., on the X-axis) and point biserial (e.g., onthe Y-axis), to divide the questions into four quadrants. The fourquadrants may be labeled with quality indicators such as poor, marginal,good, and excellent, with poor questions in the lower left quadrant ofthe scatter plot (e.g., with low percentage of students answeringcorrectly and low point biserial), marginal questions in the lower rightquadrant (e.g., with high percentage of students answering correctly andlow point biserial), good questions in the upper left quadrant (e.g.,with a low percentage of students answering correctly and a high pointbiserial), and excellent questions in the upper right quadrant (e.g.,with a high percentage of students answering correctly and a high pointbiserial).

In some implementations, the different quality-designated questions maybe presented in the scatter plots as dots of different colors toindicate quality, such as red dots indicating poor quality questions. Insome implementations, the presentation of the dots for the questions mayhave different shapes, colors, opacities, and/or other displaycharacteristics based on other criteria of the questions, which mayinclude categories of the questions. For example, the dots for questionsfrom one assessment may be presented differently compared to dots forquestions from another assessment.

In some implementations, the user may interact with the courseeffectiveness dashboard and specify the point biserial threshold thatseparates the lower portion of the scatter plot from the upper portion,and/or the percentage correct threshold that separates the left portionof the scatter plot from the right portion. For example, the left-rightthreshold may be set at 90%, and the point biserial (top-bottom)threshold may be set at 0.15. Such specification of point biserial maybe made to allow even minor improvement that is to be identified andacted upon, and/or to achieve a goal of greater variation.

The scatter plot lets a professor or other user evaluate effectivenessof questions reliability to measure student understanding and knowledge.The scatter plot also lets the user readily identify those questionsthat are of low quality and that may need to be reviewed and revised. Insome implementations, the user can click on a particular dot thatcorresponds to a question, and the question may be presented within apop-up dialog or otherwise displayed in a dashboard (e.g., the questionitself and possible answers). The dashboard may also display moredetailed psychometric data regarding the question, such as the number ofstudents who answered correctly, incorrectly, and/or with partialcredit, the distribution of selected answers (e.g., correct andincorrect), the particular students who answered correctly orincorrectly, and so forth. The dashboard can also present the history ofanswers to this question on various exams in a period of time, as wellas information regarding the assessments that include the selectedquestion.

Through the dashboard, a user can view the distribution of questions forthe assessment, and select particular questions to be removed and/oredited. In some implementations, the question can be edited or removedthrough the dashboard itself and/or the user is provided with a link tothe system that stores the question. Editing the question can includechanging the wording of the question, the wording of the possibleanswers, adding or removing possible answers, and so forth.

The dashboard can include drop-down menus and/or other selectioncontrols that let the user specify various filters and/or subsets of thedata to be displayed in the scatter plot. For example, the user canspecify a scatter plot be shown with questions that are associated witha particular category, such as a learning outcome and/or Bloom'staxonomy category. As another example, the user can specify that ascatter plot be shown with questions that are included on one or moreparticular assessments, e.g., for which the category includes one of aselected set of assessments (e.g., final exam, midterm exam, quiz number3, etc.).

In some implementations, the dashboard allows the user to graphicallyselect a set of questions by drawing a box within the scatter plot. Thedashboard can then, in response to a user command, generate a report(e.g., spreadsheet) that describes the selected questions in detail,including the psychometric data of the questions (e.g., point biserialand/or percentage of students who answered the question correctly),categories associated with the questions, and so forth. The reportand/or dashboard can also show the distribution of categories associatedwith the selected questions, such as the learning outcomes associatedwith the selected questions. For example, the user can draw a box thatapproximately corresponds to the upper right quadrant in the scatterplot, and the dashboard (and/or generated report) can display thelearning outcome categories that are associated with the selected (e.g.,excellent quality) questions.

The dashboard may also provide other data visualizations, includingtrend data showing the variation in questions of particular qualitydesignations over time (e.g., from year to year as the course istaught). In this way, the dashboard enables a user to track the changein quality of questions on particular assessment(s), and/or inparticular course(s). In some instances, the change in quality caninclude the system for comparing multiple assessment results, so thatthe difference among results (e.g., the delta) is highlighted and/orabstracted to a number/percentage improvement.

In some implementations, the course effectiveness dashboard includesfeatures that display data for students in a particular course and/orprogram of courses. The dashboard can allow a user to readily identifystudents at risk for failing a course, for example students with acourse average currently below a selected threshold (e.g., 70%) orstudents who are otherwise under-performing compared to other students.The dashboard can produce a report of such students in a particularcourse, including student names and/or ID numbers, their average scores,scores on particular assessments, which particular assessments thestudent has taken, and so forth. In some implementations, selecting(e.g., hovering over) the student in the dashboard causes the dashboardto display a distribution of scores for the selected student on thevarious assessments they have taken in a course and/or multiple courses.

In some implementations, the dashboard can apply a conditional (e.g., anif-case scenario) to identify students at risk of failing the course.For example, one exam may remain to be administered in the course, andcertain students in the course may need to earn at least a minimum scoreon the exam to achieve an overall passing grade in the course. Somestudents may be doing so poorly in the course, that no score theyachieve on the last exam will let them pass the course. Students who areclose to failing the course, and/or inevitably are to fail the course,can be designated by the dashboard as at-risk students. Otherconditionals may specify different weights for the assessments in acourse. For example, exams may be worth X % of the overall grade,quizzes may be worth Y %, and so forth. Such varying weights may beaccounted for in identifying at-risk students.

In general, the dashboard can allow the user to readily identifyproblems, and/or potential problems, in a course, and allow the user totake actions to address the problems. Such problems (e.g., risk areas)may include, but are not limited to students at risk, low qualityquestions on assessments, questions that are missing categories and/orthat have too few categories associated, learning outcomes that are notbe covered in a course and/or that are showing below threshold scores(e.g., below 70%), and so forth. In this way, the dashboard can providea visual indicator of system-evaluated problems, in which the systemautomatically determines information for further analysis, such asat-risk students and/or questions and answers that need improvement, andhighlights or otherwise identifies those students or questions. It couldalso identify categories where learning outcomes are not being correctlyassessed over time.

The dashboard can also show overall class performance on an assessmentor in a course, the number of questions on an assessment or in a course,the minimum and/or maximum scores, the distribution of scores (e.g., ina histogram), and/or various metrics, such as a metric indicating theprobability that students would obtain the same score if they took theassessment again (e.g., indicating a degree of randomness in theoutcomes of the assessment).

In some implementations, the dashboards provided by the platform canalso include an academic advising dashboard. The user of the academicadvising dashboard (e.g., a mentor) can use the dashboard to view a listof mentees (e.g., students) who have been assigned to the mentor, byentering the mapping of mentors to mentees in the UI, for mentoringactivities within the academic or other environment. The user can usethe dashboard to view data regarding their mentees, such as an averagescore of each mentee on assessments administered in their variouscourses. Mentees at risk (e.g., determined as described above) offailing one or more courses can be flagged in the dashboard, to allowthe mentor to provide help as appropriate. The user of the dashboard canselect (e.g., click on, hover over) any of the mentees shown to viewmore detailed information regarding the selected mentee, such as adistribution of assessment scores for the student, and/or for aparticular course, the score that the selected student earned on eachexam, how the student is performing with respect to learning outcomes,how the student is performance compared to class performance (e.g.,whether they are below class average or above class average), and soforth. For example, the class performance on a particular exam can becompared to the selected student's performance, and the user can performan action (e.g., hover over) in the dashboard to prompt the dashboard todisplay more detail regarding the relative performance. For example, theadditional detail may indicate that the class performance is 65% for anassessment with one question, and the particular student answered thequestion incorrectly, such that the difference between the classperformance and the student's performance is −65%. In general, questionscan be weighted differently, such that the overall score (e.g., for anindividual student or the class) is a weighted average.

The academic advising dashboard and/or other dashboards can showquestion data that is filtered according to categories, such asquestions that are associated with particular learning outcomes and/orcategories of the Bloom's taxonomy. Such filtering may help the useridentify those students that are having problems with particular typesof questions that are associated with particular categories. In someinstances, such filtering may also help identify difficultiesexperienced by students within particular groups, such as students withautism spectrum disorder (ASD). For example, the system could provideinsights into questions that might be tagged as generally more difficultfor ASD students.

In some implementations, the academic advising dashboard allows the userto enter information to memorialize various interactions between thementor and their mentees, such as meetings, counseling sessions, theoutcomes of such, and so forth. The academic advising dashboard can alsoenable the user to create CQI tasks for actions (e.g., agreed on betweenmentor and mentee) to be taken to improve mentee performance, withrespect to career development, professional certification attainment,and/or other goals.

In some implementations, the dashboards provided by the platform includea student dashboard that a particular student can log into to viewinformation regarding their performance on various assessments, withrespect to various learning outcomes, with respect to variouscategories, and so forth. The data may be displayed on the dashboardthat compares the student's performance to that of their classmates,with classmate data anonymized and/or aggregated appropriately. Thestudent dashboard can allow the logged in student to create their ownCQI tasks to track action items to improve their performance.

In some implementations, the dashboards provided by the platform includean administrator dashboard that is usable by a program administrator whooversees multiple courses and/or multiple professors within a program.Accordingly, the administrator dashboard allows the user to viewinformation regarding multiple courses and multiple professors, anddetermine how a program complies, overall, with accreditation standards.The dashboard can present information regarding compliance withaccreditation standards in a particular year or other any other suitableperiod of time. The administrator dashboard can also allow the user toview information regarding the student population within the program(e.g., demographic breakdown and/or change in breakdown over time),compare the program to other similar programs by comparing the programto peer programs at other institutions or national performance, viewcomparison information over any suitable period of time, and so forth.

In some implementations, access to the various dashboards is controlledbased on the role of the user requesting access, and permission levelsassociated with each dashboard. For example, access to informationregarding a particular course on a course effectiveness dashboard can berestricted to the professor and/or teaching assistant associated withthat course, as well as a program administrator or dean who has accessto information for multiple courses. A professor may not be able to viewinformation regarding a course that they are not associated with (e.g.,that they do not teach). For their own course(s), a professor may usethe course effectiveness dashboard to view the distribution of scoresfor assessment(s) administered in the course, and/or other informationregarding student performance in the course. A department chairperson,program administrator, dean, and/or other individual responsible foroverseeing multiple courses may use the course effectiveness dashboardto view information on the various courses that are within theirresponsibility. As described above, a particular student may use astudent dashboard to view their own scores on assessments, and thedashboard may compare their scores to the (e.g., average, anonymized,and/or aggregated) scores of other students, and the student may nothave access to other information regarding other students. Accesscontrol may be accomplished based on authentication of the user logginginto the dashboard(s) and providing credentials that are verified todetermine the access permissions of the particular user based on theirrole.

In some implementations, the platform includes, or provides anintegration with, a CQI system. The CQI system is used to generate andtrack tickets, and each ticket can describe an issue to be addressed,problem to be solved or mitigated, and/or task or action to beperformed. A CQI ticket may include information describing the issue,action(s) to be taken, and/or individual(s) who are responsible foraddressing the issue by performing the action(s). The CQI ticket mayalso indicate the current status of the ticket, such as open, closed,pending, in process, and so forth. The CQI ticket may be edited by theresponsible individual(s) to provide updates on the progress towardresolving the issue. In some implementations, the platform provides aCQI dashboard that can be used to create, view, update, close, and/orotherwise access CQI tickets. A CQI ticket is also described herein as aCQI task. The ticket can include original, current, and/or targetmetrics and states.

The various dashboards provided by the platform can be used to createand view CQI tasks associated with the data shown in the dashboard(s).The dashboard can be used to identify a problem, such as a low-qualityquestion on an assessment, a question that does not have associatedcategories, at risk student(s), and so forth. A control on the dashboardcan be used to display a dialog that lets the user add the details for aCQI ticket (e.g., description of the problem, action(s) to be taken,responsible individual(s), etc.). The dialog can include a submit buttonor other suitable control to request that the CQI ticket, including theentered information, be created. In some implementations, a screen shotof the dashboard can be attached to the CQI ticket to further illustratethe issue or problem being tracked. In some implementation, the CQIticket or observation includes a link to the dashboard that displays thedata at the time when the CQI ticket was created. Creation of the ticketcauses an email or other appropriate notification to be sent to theresponsible individual(s). The creator of the ticket may also benotified when the status and/or information in the ticket is changed.Attaching a screenshot of the dashboard to the ticket allows thatscreenshot to be compared to the dashboard after the issue has beenresolved, and enables confirmation of the appropriate issue resolution.In some implementations, a dashboard can include a time-stamp filter toenable the user to indicate the data they want to see. For example, thefilter may be used to view what the data presented in the dashboardlooked like when the ticket was created or what the data looked likebefore closing the ticket. The dashboard can provide a link with atime-stamp to the dashboard view the user was interacting with when theyrequested the creation of the ticket. The user can follow the link toview the previous dashboard view, and to easily evaluate progress bycomparing the previous view to a later view.

In some implementations, a dashboard displays a list of (e.g., currentlyopen) CQI tickets that have been created from that dashboard. Clickingon one of the listed ticket causes display of the ticket, including itsinformation and status, in the dashboard and/or in a pop-up dialog orother window. A user can close a ticket by changing its status andadding summary information describing the particular action(s) performedto resolve the problem. The user can also add a screenshot of adashboard state after resolution, to further establish that the issue isresolved. CQI tickets can be viewed and sorted based on variouscriteria, such as the creating user, responsible individuals, status,categories associated with the tickets, the course, assessment, and/orquestion associated with the ticket, and so forth.

In some implementations, the platform can automatically create CQItickets based on the application of rules to the data that is availablein the platform. For example, a periodic batch process can execute toanalyze question data for various assessments, and identify thosequestions that are not associated with any categories (or that areassociated with too few categories and/or lack appropriate categories).For those questions that are found to be insufficient with respect toassociated categories, have a low biserial value, and so forth, theprocess can automatically create a new CQI ticket that is assigned to aprofessor or other individual associated with the course in which theassessment is administered. Alternatively, the process can send anotification to the associated individual indicating the problem (e.g.,missing categories) and prompting them to create a CQI ticket to addressthe problem. In some implementations, a problem identified through theautomated data review process may prompt the creation of a CQI ticket ifan associated CQI ticket is not already open for the problem. The CQIintegration can track the timeliness and resolution of CQI tickets bystoring due dates and/or expected resolution dates for CQI tickets, andnotifying individuals if the due dates are imminent or have passed.

In some implementations, the platform provides features for gap analysisto identify gaps. For example, a curriculum of a program, including thelearning outcomes associated with the various courses and/or assessmentsin the program, may be analyzed and compared to intended curriculummapping of learning outcomes and accreditation requirements for anaccreditation organization, and unmet accreditation requirements (e.g.,gaps) in the program may be identified for investigation and resolution.Gap analysis can include extracting and integrating data for the variouscourses in a program, such as course syllabi and or assessmentsadministered in the courses. The learning outcomes extracted from thesyllabi, and/or the learning outcomes associated as categories with thequestions in the various assessments, can be aggregated and compared tothe intended curriculum mapping of learning outcomes and accreditationrequirements. Intended curriculum mapping of learning outcomes andaccreditation requirements that are not suitably covered by theaggregated learning outcomes of a course can be flagged as gaps in thecourse curriculum to be addressed by adding or modifying courses (e.g.,adding new content to a course) and/or adding or modifying content oncourse material, and assessments in a course.

In some implementations, the accreditation requirement coverage and/orother information may be presented in a curriculum effective and/or gapanalysis dashboard in the platform. The dashboard can also showinformation describing how students perform on learning outcomesassociated with board exams or other licensing type exams that arerelevant to the program. For example, the dashboard can show pass/failrates for law students on a bar exam, for those students who graduatedfrom a legal study program. The dashboard can also show studentperformance on accreditation requirements as well as learning outcomes.For example, the dashboard can include, for each student, an averagescore earned by the student on questions that are labeled with alearning outcome that corresponds to a particular accreditationrequirement, for one or more accreditation requirements. Suchassociation can be to varying degrees of specificity, according to theparticular accreditation requirements. For example, student scores canbe shown for accreditation requirements that are general (e.g., clinicalevaluation) and/or accreditation requirements that are more specific(e.g., pediatric or orthopedic clinical evaluation).

The dashboard can also show the degree of coverage for each learningoutcome and/or accreditation requirement based on how many courses,assessments, and/or questions are directed to the learning outcomeand/or accreditation requirement. The dashboard can show the various CQItickets that have been created to address any identified gaps in thecurriculum with respect to the accreditation requirements or gaps on theintended curriculum mapping of learning outcomes.

Gap analysis can also include comparing a particular program'sperformance to other programs (e.g., comparable, peer programs at otherinstitutions and/or national standards) with respect to the scoresearned by students on assessments that are relevant to accreditationrequirements and/or learning outcomes. Such a comparison may be maderelative to external data that is received from an accreditationorganization, and that is anonymized with respect to particular programand particular student. A comparison can also be made to studentachievement or activities following graduation, such as job placementrates, average salaries, and so forth.

Traditionally, schools or other educational institutions have usedstatic tools such as spreadsheets to track information internally, butsuch data tracking tends to be inconsistent and incomplete. Thesetraditional techniques also fail to provide context and are not readilyusable, given that the previously available tools are difficult to use,read, scattered, in an inconsistent format, difficult to access, and/orexhibit other shortcomings. Implementations provide an integratedplatform and portal for entry and analysis of data, tracking learningoutcomes for courses taught within a program. As described above, theplatform also provides gap analysis to identify instances in whichaccreditation requirements are not be adequately satisfied by thelearning outcomes covered in a particular program, enabling a programadministrator to readily determine whether accreditation requirementsare being met, and to easily create CQI tickets to prompt the filling ofany identified gaps in the curriculum with respect to the accreditationrequirements.

The platform receives data extracted from assessment service(s) (e.g.,ExamSoft™), to identify courses, assessments administered in courses,and categories associated with the questions on the assessments, if anycategories have been associated with the questions. Categories caninclude learning outcomes. The learning outcomes for a course can becompared to accreditation requirements for an accreditationorganization, to ensure the program is complying with accreditationrequirements.

The platform can provide for the automatic generation of reports and/orcommunication of the report(s) over one or more networks to anaccreditation agency and/or other interested parties. As a particularexample, the report can be communicated to the National Association ofBoards of Pharmacy® (NABP®) to verify that the data demonstratescompliance with the Pharmacy Curriculum Outcomes Assessment® (PCOA®).Such report(s) can include data regarding compliance with accreditationrequirements, compliance exhibits, proof of CQI tasks, improvement,tracking, and so forth. The report(s) can be requested through the UI ofthe platform.

In some implementations, the platform performs various operations foranalyzing the expected outcome coverage and results for (e.g., digitallyadministered) assessments in an educational setting and/or otherenvironments. In some implementations, a course syllabus and/or othercourse description information is analyzed to identify expected targetoutcomes (e.g., goals, objectives) to be achieved by students of thecourse, and to identify a relative weight of each of the various targetoutcomes based on their coverage in the syllabus. An assessment, such asa test, exam, or other (e.g., educational or professional) assessment,is analyzed to identify the categories that are assessed in the variousquestions and/or portions of the assessment. The target outcomes arecompared to the categories to determine the extent to which theassessment is assessing each of the target outcomes, the degree ofcorrespondence between the distribution of assessed categories and thedistribution of syllabus-indicated target outcomes, and whether anysyllabus-indicated target outcomes are not being assessed. Test resultsmay also be correlated with the target outcomes to develop metrics forsuccess/failure rate and/or rate change with regard to the variousoutcomes, for individual students as well as aggregate studentpopulations.

In some implementations, the analysis may be performed with respect toan educational or academic environment, such that the course is aneducational course taught to students within such an environment. Thecourse may involve one or more tests that are administeredelectronically using any suitable electronic testing software.Implementations provide a report that includes metrics describingwhether, and to what extent, the test(s) are assessing students withregard to categories that correspond to the target outcomes indicated bya course syllabus, such that the course is testing what it purports toteach. Such report(s) provide testing administrators or othereducational professionals access to specific metrics regarding testcoverage and student performance, and allows tests to be fine-tuned totarget the particular educational outcomes advertised in a coursesyllabus.

Previously available software products for educational assessment and/ortesting employ different data formats and/or fail to provide usefulinformation for analyzing outcomes. For example, one tool may describewhat an instructor is planning to teach in a course, without any usablemetadata for automated analysis, whereas another tool may indicate howstudents are performing, and yet another describes what instructors aredoing in their classrooms. These various tools may lack the capabilityto integrate with one another, making it difficult or impossible toclose the assessment loop and determine how effectively a course planhas been executed. Implementations described herein provide an automatedplatform to assess course outcomes based on test information and othersources of information.

In some implementations, computer software and a database is providedfor collecting, integrating, and analyzing student data before, during,and/or after the student(s) attend an academic program and/or othereducational setting. The computer software operates to provide apredictive analysis that anticipates specific actions to continuouslyadvance academic excellence in an educational setting or other academicenvironments. The computer software also provides data insights forreal-time programmatic assessment and accreditation compliance in aneducational setting or other academic environments.

In some implementations, the computer software (e.g., application(s))described herein automatically integrate and analyze data from disparatekey data systems in the field of education. In some instances, thecomputer software is provided on-line, such that the software is hostedand provided as a service, and can be described as software-as-a-service(SaaS). Some implementations provide a website featuring technology thatenables users to automatically integrate and analyze data from disparateassessment systems in the field of education. The software also providespredictive analysis that anticipates specific actions to continuouslyadvance academic excellence in the field of education. Implementationsalso provide a user interface (UI) featuring technology that enablesusers to conduct predictive analysis that anticipates specific actionsto continuously improve an educational program's quality. The softwarecan provide data insights needed for real-time programmatic assessmentand accreditation compliance in the field of education. Someimplementations provide a website that features technology that enablesusers to display proprietary data insights for real-time programmaticassessment and accreditation compliance in the field of education.Implementations also provide (e.g., on-line, non-downloadable) computersoftware that documents data-driven decisions to continuously advanceacademic excellence.

Implementations provide a process that automatically extracts,transforms, normalizes, and integrates data from key data sources.Implementations also provide a secured data warehouse that automaticallystores and processes the extracted data to deliver meaningful datathrough interactive dashboards and/or an application programminginterface (API). Moreover, implementations also include interactivedashboards that provide a view of academic assessment data (e.g.,student-level data) and/or institutional data (e.g., operationalperformance data) for educational program improvement, accreditationcompliance, and predictive analytics. Such dashboards can helpinstitutions make better strategic decisions and improve its operations.

Institutional data can include, but is not limited to, data describingone or more of the following: student retention, student attrition,standardized survey responses, national examination grades,application/admissions data, and/or post-graduation data (e.g., jobplacement, alumni events, etc.).

Academic assessment data can include, but is not limited to, datadescribing one or more of the following: national examinations,admissions data, electronic assessments, academic infractions,attendance, simulation activities, co-curricula activities, clinicalactivities (e.g., clerkships, rotations, etc.), community service, boardexams, and/or standardized exams.

In some implementations, the dashboards are included (e.g., embedded) ina UI that contains a Data-Driven Continuous Quality Improvement formstask tracking system that allows users to add real-time comments and/oradditional insights related to a specific data view. In these forms, theusers can create, update, and complete Continuous Quality Improvement(CQI) tasks. Creation of a CQI task can include one or more of thefollowing.

When creating a CQI task, the user can add a data insight to interpretand communicate the data analysis of what is going on with a particulardata view.

When creating a CQI task, the user can add a Data-Driven Decision, aquality improvement decision based on the data insights, and/orcontinuous data-based actions that lead to measurable qualityimprovements of academic degree programs.

When creating a CQI task, the user can specify the desired outcome ofthe Data-Driven Decision in quantitative terms in the goal field.

When creating a CQI task, the user can assign a CQI tasks to an assigneethat is a professor, committee, or course.

When creating a CQI task, the user can set a date for re-evaluationand/or completion of the task.

When creating a CQI task, the user can label the CQI task withcategories. Examples of categories include mission, institutional goals,strategic plan goals, accreditation standards, and so forth.

After a CQI task is submitted, the creator and assignor are sent aconfirmation (e.g. email), and the data is sent to the database alongwith a screenshot of the data view. In some instances, the assignee canupdate and complete the CQI task. When updating the CQI Task, theassignee can specify the status, the success of the data driven decisionand add a comment/insight.

When a CQI task is completed and/or closed, the assignee and creator aresent a confirmation (e.g. email). When closing and/or completing the CQItask, the data is sent to the database along with a screenshot of thedata view. The assignee and creator can be sent a reminder (e.g., email)if the CQI is not closed and/or completed before thereevaluation/completion date. In some implementations, the system takesa screenshot of the data view and sends it to the database, if the CQIis not closed and/or completed before the reevaluation/completion date.

The creator can monitor the status of CQI Tasks. The CQI tasks can bereported by assigned labels to identify successful decisions before andafter the Data Driven Decision. For example, a report can be generatedthat shows all the CQI tasks related to Standard 3 of the AccreditationStandards.

In some implementations, the UI includes forms to capture data thatschools or other educational organizations would like to integrate withkey data systems. These forms alleviate the need for schools themselvesto keep track of institutional and/or academic data in the traditionalmanner (e.g., using spreadsheets, etc.). Example forms can be providedfor curriculum map, co-curricula activities, academic infractions,mentor/mentee relationships, and so forth.

FIG. 1 depicts an example system 100 for assessment data analysis,according to implementations of the present disclosure. As shown in theexample, the system 100 can include one or more analysis devices 102,which host and provide the assessment data analysis platform (alsoreferred to as the platform). The analysis device(s) 102 can include anysuitable number and type of computing device(s), and may includedistributed computing device(s) (e.g., cloud computing server(s)). Theanalysis device(s) 102 may include an analysis engine 104 that performsthe various types of analysis as described herein to generate analysisresults 118. The analysis engine 104 may receive, as input, learningoutcome data 108 generated based on an analysis and/or examination ofone or more course syllabi 106, course description(s), and/or otherinformation describing a course (e.g., a university class, trainingseminar, etc.). The outcome data 108 for a particular course may includeone or more target learning outcomes that are purported to be taughtwithin a course. The learning outcome data 108 may be associated withone or more courses. A target outcome may be particular to the type ofcourse. For example, target outcomes for a beginning calculus course mayinclude: learn how to determine a derivative, learn how to determine anintegral, understand sequences and series, and so forth. As anotherexample, target outcomes for a first aid course may include: learn howto perform CPR, learn how to treat a simple wound, learn how to treathypothermia, and so forth. Outcome data 108 for a course may bespecified, e.g., by an instructor, in at least one course syllabus 106.In some instances, the outcome data 108 may be determined through anatural language (NL) analysis or other type of analysis that isperformed on the syllabus or other course materials. In some instances,the learning outcome data 108 may be entered, e.g., by a professor orother individual, through the UI provided by the platform. The entereddata can be reflected in the dashboards hosted by the UI.

In some implementations, the outcome data 108 may indicate a weight foreach outcome, describing how each target outcome is to be emphasizedwithin a course. For example, a course may include target outcomes X, Y,and Z with equal weights of 33%, indicating that each outcome is to betaught for a substantially equal amount of time or emphasizedsubstantially the same. As another example, a course may include targetoutcomes X, Y, and Z with weights 10%, 30%, and 60% respectively,indicating the different relative emphasis to be placed on each outcomeduring teaching of the course.

The analysis engine 104 may communicate, over one or more networks, withone or more assessment services 110. The assessment service 110 may bean online service and/or software application that provides educationalor other types of test, and that may store and provide informationregarding administered assessment(s) 112. In some implementations, theassessment service 110 is an electronic assessment (e.g., testing)software package, such as that provided by ExamSoft™. The assessmentservice 110 may provide, to the analysis engine 104, assessment data 140that includes category data 114 and score data 116. The category data114 may identify one or more categories that are assessed by orotherwise associated with questions in each of the assessment(s) 112.For example, a particular assessment 112 for a calculus course may testderivatives, integrals, and so forth, such that a question may be taggedwith a category of derivative, integral, etc. In some instances, thecategories may be from the same set of outcomes as described in theoutcome data, in instances where the category data 114 and outcome data108 is normalized to describe topics (e.g., outcomes and categories)using a same set of possible topics. Alternatively, the categories maybe described differently than the outcomes, such that the category data114 is not normalized with the outcome data 108 in the assessment data140 and/or outcome data 108 that is received by the platform. In suchinstances, the analysis engine 104 may perform a normalization of theinput data prior to further analysis. Such normalization may includedetermining a mapping between outcomes and categories, and/ordetermining mappings between outcomes, categories, and a common set oftopics that describe both outcomes and categories.

In some implementations, the category data 114 includes weights ofvarious categories that are assessed by an assessment 112, as describedabove. For example, if a history test includes 20 questions, and 3 ofthe questions assess knowledge of the American Civil War, and 5 of thequestions assess knowledge of the Reconstruction period of Americanhistory, the category data 114 may indicate that the test assesses the“American Civil War” category with a weight of 3/20, or 15%, and thetest assesses the “Reconstruction” category with a weight of 5/20, or25%. The category data 114 may be specified by a course instructor orother individual, and/or may be determined through a NL analysis orother type of analysis of the questions in one or more assessments 112.

In some implementations, the assessment service 110 may also providescore data 116 to the analysis engine 104. The score data 116 maydescribe, for each of one or more tests, the overall score(s) ofstudent(s) on the test, as well as the scores(s) of student(s) onindividual questions of the test. The assessment data 140 may alsoinclude a description of the assessment(s) 112 themselves, including thequestions on each assessment, possible answers to each question, weightsof the questions, and so forth. The assessment data 140 can also includeother information, such as the course in which an assessment isadministered, the date on which the assessment was (or is scheduled tobe) administered, a professor and/or other individual(s) associated withthe course, and so forth. Assessment data 140 can also include, for eachquestion, psychometric data such as a point biserial, percentage ofstudents who answered the question correctly, and/or other informationas described above.

The outcome data 108 and the assessment data 140 may be analyzed by theanalysis engine 104 to generate analysis results 118, as describedabove. Such analysis may include a gap analysis, such as comparing theoutcome data 108 to the category data 114 to determine whether all theoutcomes that the course purports to cover are actually being tested bycategories in the assessment(s) 112 for the course. For those outcomesthat have corresponding categories covered by the test(s), theassessment information may indicate whether the proportions or weightsof the outcomes corresponds to the proportions or weights of thecategories. For example, the analysis results 118 may indicate that aparticular target outcome in the outcome data 108 for a course does nothave any corresponding categories in the category data 114 for theassessment(s) 112 in that course. As another example, the analysisresults 118 may indicate that the outcome data 108 lists, as targetoutcomes, A, B, and C, with weights 50%, 25%, and 25% respectively,whereas the category data 114 indicates that A is tested only withweight 20%, B is tested more heavily with weight 70%, and C is nottested at all.

The analysis results 118 may be presented through one or moreinteractive dashboards 126 as described herein, such as a courseeffectiveness dashboard, academic advising dashboard, administratordashboard, student dashboard, CQI integration dashboard, gap analysisdashboard, and so forth. A user 128 may access the interactivedashboard(s) 126 using a user device 124 of any suitable type. Asdescribed above, in some instances, access to a dashboard 126 and/oranalysis results 118 presented within a dashboard is limited to thoseusers 128 who have permission to access the information, based on theuser role. For example, a particular professor may be able to view datafor their own course(s) but not for other course(s) taught by otherprofessors. As another example, a student may be able to view dataregarding their own performance, as compared to anonymized, aggregateddata for other students, but may not be able to view data regarding theperformance of other identified students.

In some implementations, the interactive dashboards 126 includecontrol(s) that allow a user to request the generation of report(s) 122that describe the analysis results 118. Such report(s) 122 can begenerated through a reporting interface 120 that executes within theplatform or elsewhere. The reporting interface 120 can output one ormore reports 122. The report(s) 122 may be communicated to userdevice(s) 124 for presentation in a user interface (UI) on the userdevice(s) 124. The report(s) 122 may also be stored for future access.The report(s) 122 may include analysis results 118 in various forms,including but not limited to tabular data, graphs of various types,animated graphics, textual descriptions of assessment(s), and so forth.Examples of analysis results 118 shown in the dashboards 126 and/orprovided through the reports 122 is described further with reference toFIGS. 4-10.

In some implementations, the analysis engine 104 includes one or morecorrelation module(s) 130 that perform an analysis to correlate outcomedata 108 with assessment data 140. Such analysis can be performed togenerate analysis results that include course effectiveness data, suchas that presented through the course effectiveness dashboard, asdescribed herein. The analysis engine 104 can also include gap analysismodule(s) 132 that perform the gap analysis as described above. The gapanalysis can produce analysis results 118 that describe gaps in a coursecurriculum compared to accreditation requirements. The accreditationrequirements can be received as other data 138 from one or more externaldata sources 136, such as an accreditation organization. Other types ofother data 138 may also be received and analyzed by the platform. Theanalysis engine 104 may also include a CQI integration module 134 toenable CQI tickets (tasks) to be created, updated, and tracked throughthe dashboards 126, as described above.

In some implementations, the analysis engine 104 includes anormalization module 142 to perform normalization operations asdescribed herein to normalize the range of available credit to beconsistent (e.g., a range from 0 to 1) across multiple questions thatmay each originally have different maximum score values. Normalizationoperations performed by the normalization module 142 as describedfurther herein. Categories may also be normalized. For example, eachschool can have its own category structure and the platform cannormalize the categories to a set of standardized categories that areapplicable across data from multiple schools or other organizations.Normalization is further described with reference to FIG. 2C.

To assist an organization in meeting its accreditation requirements, orother types of requirements that may be imposed by external entities asdescribed herein, the CQI integration module 134 can provide a ticketingsystem that tracks and presents information relative to accreditationrequirements and/or that tracks and presents information at differentpoints in time to track the organization's progress in meeting theiraccreditation requirements. For example, the user of the platform canview a dashboard that shows a particular data state, and use the ticketgeneration function of the dashboard to automatically generate a ticketthat is tracked in the CQI system. The generated ticket can include asnapshot of the dashboard, or at least a portion of the dashboard, whichshows the presented data that indicates the problem to be solved, e.g.,the problem that is tracked through the ticket. On resolution of theticket, the dashboard snapshot can be compared to a correspondingdashboard snapshot at a later time/date, to check whether the“problematic data” has been resolved, at least in part. For example, ifthe dashboard shows that a particular accreditation goal (or other goal)is not being met, a ticket can be generated that includes a snapshot ofthe dashboard illustrating the unmet goal. A later resolution of theticket (e.g., closing the ticket) can include attaching, to the resolvedticket, a later snapshot of the same dashboard showing that the goal hasbeen met. In some implementations, the user makes certain selectionsthrough the dashboard to generate the captured state of the dashboard,and those selections can also be captured and associated (or added to)the generated ticket, to enable the corresponding dashboard state to belater generated for comparison, based on the same user selections.

For example, the user of the platform can use the dashboard to create aticket for a task to be performed, the ticket being tracked through theticketing system. The ticket can also include a copy of the query thatthe user performed (e.g., the selections made by the user), in thedashboard, to instruct the dashboard to generate the particulardashboard state, which is added as a snapshot to the ticket. Thesnapshot (also described as a screenshot) of the dashboard is stored inthe ticket, along with the user selections made to generate the capturedstate of the dashboard. When the ticket is closed, the platform cangenerate change information (e.g., a delta) that describes the change inthe data from when the ticket was generated compared to the currentstate of the data, where the current state of the data is determined byrunning the same query (e.g., the user selections) that were previouslyused to generate the previous state. In this way, implementationsprovide a reliable comparison of previous data state to current datastate, to enable tracking to determine whether a problem has beenresolved, such as whether accreditation requirements or other goals havebeen achieved. The platform also can provide a feature to let the userview the previous state of the dashboard (e.g., when the ticket wascreated) in comparison to the current state, to visually examine how thedata has changed over time.

The dashboard can also let the user request creation of a ticket thatidentifies a particular data point, such as a particular student,question, and/or other data element, that is to be addressed throughresolution of the ticket. The platform can automatically generate aticket to resolve the issue, based on the context of the data point(s)that have been identified by the user as problematic. The user canspecify a deadline for resolution of the issue, in some instances,and/or a particular individual who is assigned to perform task(s) toresolve the issue.

The platform also enables a user to request that a report be generatedthat lists tasks that are being tracked through the ticketing system.The report can be requested to show tasks that have been completed(e.g., resolved) over a particular period of time, tasks that are notyet resolved, and/or tasks that are in some other status (e.g., open,pending, resolved, etc.). The report can also list tasks assigned toparticular individual(s). The reporting system can accept a query from auser, specifying criteria for the generated report such as status,date/time range (e.g., date/time when the tickets were created orclosed), assigned individual(s), and so forth. The generated report caninclude some or all of the data fields that are included in the ticket,and the query can indicate which data fields are to be presented as partof the report. For example, the query can indicate whether the dashboardscreenshots are to be included in the report. The report can alsoinclude change information that indicates the previous value of thetracked data point (e.g., a particular exam score) compared to thecurrent value of the tracked data point. For different tickets,different data points may be relevant to the particular issues beingtracked by the tickets, and the change information can also reflect thedifferent relevant data points. On closing a ticket, the system canattach the current dashboard screenshot showing the data state when theticket is closed, to provide a comparison between the state when theticket was opened and the corresponding state when the ticket wasclosed. In closing a ticket, the user can indicate whether the issue wasresolved, how the issue was resolved (if resolved), why the issue wasnot resolved (if not resolved), progress made toward resolving theticket, and/or other suitable information regarding the closing of theticket. A follow-up ticket can also be created to describe additionaltasks to be performed, the follow-up ticket associated with the originalticket and in some instances inheriting at least a portion of theinformation included in the original ticket (e.g., with respect to theissue to be resolved, responsible entities, and so forth).

FIG. 2A depicts an example of outcome data 108 that may be employed inoutcome correlation analysis, according to implementations of thepresent disclosure. The outcome data 108 may describe a single course,or multiple courses. For each course, the outcome data 108 may includeany appropriate number of target outcomes 202, also described aslearning outcomes. A record for a target outcome 202 may include anoutcome tag 204 that describes the outcome, and a weight 206 thatdescribes the proportional importance or emphasis of that particularoutcome among the total set of outcomes identified for the course.

FIG. 2B depicts an example of category data 114 that may be employed inoutcome correlation analysis, according to implementations of thepresent disclosure. The category data 114 may describe categoriesassigned to questions of a single assessment 112, or multipleassessments 112. For each assessment, the category data 114 may includeany suitable number of records of question data 208, each recordassociated with a particular question on the test. A question datarecord may include any number of category tags 210, identifying theparticular categories that are assessed by that question on the test.

The outcome tags and categories tags may also be described as metadatatags, that provide a description of the identified outcomes andcategories respectively.

In some instances, the analysis results are determined based on ananalysis of input data from multiple sources that include course syllabior other forms of course descriptions, as well as outcome from thetesting service (e.g., electronic testing software package(s)). In someimplementations, the analysis results describe whether, and how well, anassessment for a course is covering the various target outcomes thatwere indicated by the course syllabus or other course descriptionmaterials. In this way, the platform can evaluate whether a course isassessing on what it purports to teach. Faculty may tag variousquestions in an assessment with categories when they write theassessment and/or upload it to the assessment service, by specifying alearning outcome, indicating a chapter from a textbook, or through useof other types of category tags that describe a topic, category, and/orpurpose of a question. Such tags may be compared, by the analysisengine, to the course syllabus to assess a degree of correlation betweensyllabus and assessment. For example, if the syllabus claims thatstudents are to be taught three target outcomes during a course, theassessment may determine whether the questions on the assessmentactually targeted those three target outcomes, or if the assessmentcovered other categories instead. Gaps in the test coverage of thetarget outcomes, and/or other discrepancies between target outcomes andtested categories may be flagged as problems in the reports that areoutput from the system.

FIG. 2C depicts a flow diagram of an example process for normalization,according to implementations of the present disclosure. Operations ofthe process can be performed by the normalization module 142, and/orother software module(s) executing on the platform or elsewhere.

Raw score data and/or category data is received (220). The raw scoredata and/or category data is normalized (222). The normalized score dataand/or category data is provided for subsequent analysis and processingby the platform as described herein.

In some examples, normalization of score data can include modifying thescore data to be within a consistent scale across various assessments,courses, and/or institutions. For example, in one course, a professormay score exam questions on a scale of 1 to 10, where 1 is minimumcredit and 10 is maximum credit. In another course, a professor mayscore exam questions on a scale from 0 to 20, where 0 is minimum creditand 20 is maximum credit. Normalization of the score data can includeadjusting each score to be within a consistent range (e.g., from 0 to1). For example, if a raw data score was 5 on the scale from 0 to 20,the normalized score may be 0.25 on a scale from 0 to 1. Implementationscan normalize score data to be within any suitable range of values.Normalization can also include applying a numeric score to raw datascores that were originally described more qualitatively. For example,raw data scores can be assigned to one of five qualitative values:“failing,” “below average,” “average,” “above average,” and “exceedsexpectations.” Normalization can transform these values to 0, 0.25, 0.5,0.75, and 1.0 respectively, be within the range from 0 to 1.

Normalization of category data can include the mapping of inputcategories to a consistent set of categories that is applied across theassessment data from any number of assessment services. For example, oneassessment service may provide assessment data that includes a categoryfor “integral calculus,” and another assessment service may provideassessment data that includes a category for “integrals.” Normalizationmay place data that has been categorized into “integral calculus” and“integrals” respectively into a same normalized category “integration.”Accordingly, the platform can analyze the semantic differences incategorization and impose a standard set of categories that can beconsistently applied across assessment data from any number of sources.

The score data may also be correlated with target outcomes and/orcategories to assess student performance on an assessment with respectto the various tested categories. For example, the platform maydetermine how well each student (or a class of students) performed onthose questions that targeted the particular categories that correspondto particular target outcomes. In this way, the platform may determinehow well a course did in teaching to its indicated target outcomes. Thisanalysis also enables a more fine-grained assessment of studentperformance on outcomes. For example, the student may receive an overallgrade in a calculus course, as well as grades or other performanceevaluations in sub-topics corresponding to the outcomes and/orcategories covered in the course.

As described above, in instances where the outcomes and categories arelabeled inconsistently relative to each other, or inconsistently acrossmultiple courses and/or assessments, the analysis engine 104 maydetermine a mapping that provides a consistent set of tags for use indescribing the outcomes and categories. For example, an assessment mayassess topics that were taught in multiple courses (e.g., for a boardcertification exam, licensing exam, comprehensive exam, etc.), and theanalysis may employ the same set of tags across all the courses and forthe assessment as well. Consistency in outcome and/or category tagsenables evaluation of student performance and/or course quality over awhole education of multiple courses, not limited to a particular course.

In some implementations, the analysis may include determination of thequality of questions on one or more assessments. In such instances, theanalysis engine may access psychometric data for the questions in anassessment, and use the data to evaluate question quality to determinewhether questions may be improved. As described above, question qualitymay be determined based on a comparison of the point biserial of aquestion to the percentage of students who answered the questioncorrectly.

In some implementations, the analysis may assess how students areperforming on learning outcomes that are mapped with the categories usedin the assessment(s). For example, if an accreditation agency requiresparticular outcomes for a student to be accredited or licensed for aparticular profession or specialty, the system can map labels to theaccreditation-relevant external outcomes, and assess coverage via themapping (e.g., determine whether a student satisfied accreditationrequirements).

For each course to be analyzed, the analysis engine may collect andanalyze data from the course syllabus and/or other materials todetermine the target outcomes to be covered by each course. The categorydata may be received from the assessment service, in the form of one ormore exports of data sets per assessment administered by the assessmentservice. Such exports may include student scores on the assessments, aswell as scores on individual questions of the assessments. The studentscore data may be anonymized, aggregated, and/or obfuscated to ensureprivacy. The exported data sets may also include the categories for eachquestion (e.g., as provided by the professor or automatically determinedthrough NL analysis). The exports may also include psychometric data forquestions. The categories are compared to the target outcomes for acourse to determine the degree of coverage of the assessment, e.g.,whether the assessed categories on the questions of the assessmentcorrespond to the target outcomes.

Score data may be integrated with outcome data to determine studentperformance per outcome. The analysis may identify any gaps in coverageof the assessment. The analysis may also identify any gaps ordeficiencies in student performance with respect to one or moreparticular outcomes, where such outcomes are assessed. The outputreports may describe the analysis results for a particular assessmentand/or for a course as a whole (e.g., including multiple assessments).Reports may indicate one or more of the following: gaps in coverage;whether the correct weight or proportion of learning outcomes is beingtested; whether there are any unlabeled questions in an assessment(e.g., questions without any indicated categories); how studentsperformed with respect to particular outcomes, where category data isavailable to make such a determination; the quality of questions, basedon psychometric data; and/or score distributions for questions.

Reports can document everything that is provided in the dashboards alongwith the task fields and comments, or at least some portion thereof.Reports may also indicate how many students got a question correct orincorrect, and whether those who answered correctly or incorrectly arehigher performers or lower performers in a student population. Suchinformation may be used to infer question quality, by comparing thepoint biserial of a question to the percentage who answered the questioncorrectly, as described above. For example, if students who answered aquestion correctly are primarily those who did poorly on the assessment,or are similar in proportion to those students who did well on theassessment, as indicated by the point biserial, that might indicate thatthe correctly answering students were guessing on the question,indicating a potentially low quality for the question. If percentage ofstudents answering correctly correlates with those students who did wellon the assessment, that may indicate a high-quality question. Asdescribed above, the course effectiveness dashboard may graphicallypresent this comparison in a scatter plot, with question qualityindicated by the position of the question in the scatter plot.

Reports may present assessment data in various forms. In some instances,a report may present a pie chart showing a percentage of learningoutcomes per course and per assessment. A report may also present acorrelation of a course, or an assessment, with the various categoriesof Bloom's taxonomy—a set of hierarchical models used to classifylearning objectives into levels of complexity and/or specificity. Forexample, a knowledge-based (e.g., cognitive) model may include ahierarchy of remembering (e.g., a lowest level), comprehending,applying, analyzing, synthesizing, and evaluating (e.g., a highestlevel). The reports may evaluate based on criteria that higher levelcourses (e.g., advanced courses) should provide learning at a higherlevel in the hierarchy than lower-level courses (e.g., introductorycourses). Questions on an assessment may be tagged with indication ofthe taxonomy level associated with the question, or taxonomy level maybe determined automatically based on NL analysis. Courses, assessments,and/or particular questions may be evaluated with regard to whether theyare assessing at the appropriate level in the taxonomy.

FIG. 3 depicts a flow diagram of an example process for assessment dataanalysis, according to implementations of the present disclosure.Operations of the process may be performed by one or more of theanalysis engine 104, reporting interface 120, the correlation module(s)130, the gap analysis module(s) 132, the CQI integration 134, theinteractive dashboard(s) 126, and/or other software executing on theanalysis device(s) 102, the user device(s) 124, or elsewhere.

As described above, learning outcome data is accessed (302) andassessment data is accessed (304). The analysis results are generated(306) based on an analysis of the assessment data with respect to theoutcome data. For example, a comparison of the outcome data and thecategory data may be performed to determine a degree of correlationbetween the target outcomes and the assessed categories for a course.Other assessments may also be performed based on score data,psychometric data, and/or other information as described above. Theanalysis results may be presented (308) through interactive dashboards,as described herein. The analysis results may also be provided (310) inone or more reports that are sent to report consumers and viewed on userdevices.

In some implementations, the analysis results may be provided throughinteractive dashboards that allow a user to interactively query andrefine the information that is being presented through the dashboards.FIGS. 4-10 depict examples of analysis results 118 and other informationpresented through dashboard(s), according to implementations of thepresent disclosure.

FIG. 4 provides an example of an item analysis scatter plot forevaluating questions. The x-axis tracks, for each question, a number orproportion of students who correctly answered the question. The y-axistracks a point biserial for each question, indicating a correlationbetween correctly answering students and their level of performance(e.g., high performers on the test versus low performers). The locationof each question on the scatter plot may indicate a quality of thequestion, e.g., whether a correct answer on the question correlates(e.g., high quality question) or does not correlate (e.g., low qualityquestion) with high-performing students.

FIG. 5 provides an example of a chart that may be used to identify thosestudents at risk of academic failure, based on the assessment output.

FIG. 6 provides an example of a chart indicating the level of studentachievement with respect to target learning outcomes.

FIG. 7 provides an example of a chart indicating the level of studentachievement with respect to learning outcomes for national competencies.

FIG. 8 provides an example of a chart indicating the level of studentachievement on various target learning outcomes that are subjects in oneor more courses.

FIG. 9 provides an example of a student report card showing, for aparticular student, their level of achievement with regard to particularoutcomes, nationally tracked competencies, and subjects in course(s).

FIG. 10 provides an example of an analysis of curriculum gap andcurriculum effectiveness, indicating a level of correspondence betweentested categories and target course outcomes, as well as any gaps wherethe test(s) failed to assess performance with regard to any targetoutcomes of a course.

The U.S. Department of Education and various national agencies for theaccreditation of programs in schools of health professions haveestablished standards that are to be met or exceeded by the schools toensure that graduates are prepared to enter practice. The schools arerequired to implement a plan to assess attainment of educationaloutcomes. The plan needs to measure student achievement at definedlevels of the professional competencies that support attainment of theeducational outcomes in aggregate and at the individual student level.Schools are also required to systematically assess the curricularstructure, content, organization, and outcomes, and use assessment datafor continuous improvement of the curriculum and its delivery.Traditionally, schools dedicate significant time and resourcesdeveloping and planning assessments, with the aim of measuring andrecording the results of desired outcomes. Less time is spent convertingthe data to generate actionable information and insights forprogrammatic assessment, accreditation compliance, and predictiveanalytics. Implementations described herein provide efficient andautomatic academic assessment tools, and provide intuitive dashboardsthat show data insights for programmatic assessment and accreditationcompliance. The system automatically integrates and analyzes data fromdisparate assessment systems, and employs predictive data science toanticipate specific actions to improve the quality of an educationalprogram. The system helps students achieve educational outcomes andtrack their progress along the way, and helps educational institutionsachieve and maintain compliance with accreditation standards by usingreal-time data. The system enables a program to identify which studentsare on track to pass their board certification or other exams, and whichmay need intervention and assistance.

FIG. 11 depicts an example computing system, according toimplementations of the present disclosure. The system 1100 may be usedfor any of the operations described with respect to the variousimplementations discussed herein. For example, the system 1100 may beincluded, at least in part, in one or more components of the system 100,such as the user device 124, the analysis device(s) 102, and/or othercomputing device(s) or computing system(s) described herein. The system1100 may include one or more processors 1110, a memory 1120, one or morestorage devices 1130, and one or more input/output (I/O) devices 1150controllable via one or more I/O interfaces 1140. The various components1110, 1120, 1130, 1140, or 1150 may be interconnected via at least onesystem bus 1160, which may enable the transfer of data between thevarious modules and components of the system 1100.

The processor(s) 1110 may be configured to process instructions forexecution within the system 1100. The processor(s) 1110 may includesingle-threaded processor(s), multi-threaded processor(s), or both. Theprocessor(s) 1110 may be configured to process instructions stored inthe memory 1120 or on the storage device(s) 1130. For example, theprocessor(s) 1110 may execute instructions for the various softwaremodule(s) described herein. The processor(s) 1110 may includehardware-based processor(s) each including one or more cores. Theprocessor(s) 1110 may include general purpose processor(s), specialpurpose processor(s), or both.

The memory 1120 may store information within the system 1100. In someimplementations, the memory 1120 includes one or more computer-readablemedia. The memory 1120 may include any number of volatile memory units,any number of non-volatile memory units, or both volatile andnon-volatile memory units. The memory 1120 may include read-only memory,random access memory, or both. In some examples, the memory 1120 may beemployed as active or physical memory by one or more executing softwaremodules.

The storage device(s) 1130 may be configured to provide (e.g.,persistent) mass storage for the system 1100. In some implementations,the storage device(s) 1130 may include one or more computer-readablemedia. For example, the storage device(s) 1130 may include a floppy diskdevice, a hard disk device, an optical disk device, or a tape device.The storage device(s) 1130 may include read-only memory, random accessmemory, or both. The storage device(s) 1130 may include one or more ofan internal hard drive, an external hard drive, or a removable drive.

One or both of the memory 1120 or the storage device(s) 1130 may includeone or more computer-readable storage media (CRSM). The CRSM may includeone or more of an electronic storage medium, a magnetic storage medium,an optical storage medium, a magneto-optical storage medium, a quantumstorage medium, a mechanical computer storage medium, and so forth. TheCRSM may provide storage of computer-readable instructions describingdata structures, processes, applications, programs, other modules, orother data for the operation of the system 1100. In someimplementations, the CRSM may include a data store that provides storageof computer-readable instructions or other information in anon-transitory format. The CRSM may be incorporated into the system 1100or may be external with respect to the system 1100. The CRSM may includeread-only memory, random access memory, or both. One or more CRSMsuitable for tangibly embodying computer program instructions and datamay include any type of non-volatile memory, including but not limitedto: semiconductor memory devices, such as EPROM, EEPROM, and flashmemory devices; magnetic disks such as internal hard disks and removabledisks; magneto-optical disks; and CD-ROM and DVD-ROM disks. In someexamples, the processor(s) 1110 and the memory 1120 may be supplementedby, or incorporated into, one or more application-specific integratedcircuits (ASICs).

The system 1100 may include one or more I/O devices 1150. The I/Odevice(s) 1150 may include one or more input devices such as a keyboard,a mouse, a pen, a game controller, a touch input device, an audio inputdevice (e.g., a microphone), a gestural input device, a haptic inputdevice, an image or video capture device (e.g., a camera), or otherdevices. In some examples, the I/O device(s) 1150 may also include oneor more output devices such as a display, LED(s), an audio output device(e.g., a speaker), a printer, a haptic output device, and so forth. TheI/O device(s) 1150 may be physically incorporated in one or morecomputing devices of the system 1100, or may be external with respect toone or more computing devices of the system 1100.

The system 1100 may include one or more I/O interfaces 1140 to enablecomponents or modules of the system 1100 to control, interface with, orotherwise communicate with the I/O device(s) 1150. The I/O interface(s)1140 may enable information to be transferred in or out of the system1100, or between components of the system 1100, through serialcommunication, parallel communication, or other types of communication.For example, the I/O interface(s) 1140 may comply with a version of theRS-232 standard for serial ports, or with a version of the IEEE 1284standard for parallel ports. As another example, the I/O interface(s)1140 may be configured to provide a connection over Universal Serial Bus(USB) or Ethernet. In some examples, the I/O interface(s) 1140 may beconfigured to provide a serial connection that is compliant with aversion of the IEEE 1394 standard.

The I/O interface(s) 1140 may also include one or more networkinterfaces that enable communications between computing devices in thesystem 1100, or between the system 1100 and other network-connectedcomputing systems. The network interface(s) may include one or morenetwork interface controllers (NICs) or other types of transceiverdevices configured to send and receive communications over one or morecommunication networks using any network protocol.

Computing devices of the system 1100 may communicate with one another,or with other computing devices, using one or more communicationnetworks. Such communication networks may include public networks suchas the internet, private networks such as an institutional or personalintranet, or any combination of private and public networks. Thecommunication networks may include any type of wired or wirelessnetwork, including but not limited to local area networks (LANs), widearea networks (WANs), wireless WANs (WWANs), wireless LANs (WLANs),mobile communications networks (e.g., 3G, 4G, Edge, etc.), and so forth.In some implementations, the communications between computing devicesmay be encrypted or otherwise secured. For example, communications mayemploy one or more public or private cryptographic keys, ciphers,digital certificates, or other credentials supported by a securityprotocol, such as any version of the Secure Sockets Layer (SSL) or theTransport Layer Security (TLS) protocol.

The system 1100 may include any number of computing devices of any type.The computing device(s) may include, but are not limited to: a personalcomputer, a smartphone, a tablet computer, a wearable computer, animplanted computer, a mobile gaming device, an electronic book reader,an automotive computer, a desktop computer, a laptop computer, anotebook computer, a game console, a home entertainment device, anetwork computer, a server computer, a mainframe computer, a distributedcomputing device (e.g., a cloud computing device), a microcomputer, asystem on a chip (SoC), a system in a package (SiP), and so forth.Although examples herein may describe computing device(s) as physicaldevice(s), implementations are not so limited. In some examples, acomputing device may include one or more of a virtual computingenvironment, a hypervisor, an emulation, or a virtual machine executingon one or more physical computing devices. In some examples, two or morecomputing devices may include a cluster, cloud, farm, or other groupingof multiple devices that coordinate operations to provide loadbalancing, failover support, parallel processing capabilities, sharedstorage resources, shared networking capabilities, or other aspects.

Implementations and all of the functional operations described in thisspecification may be realized in digital electronic circuitry, or incomputer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them. Implementations may be realized asone or more computer program products, i.e., one or more modules ofcomputer program instructions encoded on a computer readable medium forexecution by, or to control the operation of, data processing apparatus.The computer readable medium may be a machine-readable storage device, amachine-readable storage substrate, a memory device, a composition ofmatter effecting a machine-readable propagated signal, or a combinationof one or more of them. The term “computing system” encompasses allapparatus, devices, and machines for processing data, including by wayof example a programmable processor, a computer, or multiple processorsor computers. The apparatus may include, in addition to hardware, codethat creates an execution environment for the computer program inquestion, e.g., code that constitutes processor firmware, a protocolstack, a database management system, an operating system, or acombination of one or more of them. A propagated signal is anartificially generated signal, e.g., a machine-generated electrical,optical, or electromagnetic signal that is generated to encodeinformation for transmission to suitable receiver apparatus.

A computer program (also known as a program, software, softwareapplication, script, or code) may be written in any appropriate form ofprogramming language, including compiled or interpreted languages, andit may be deployed in any appropriate form, including as a standaloneprogram or as a module, component, subroutine, or other unit suitablefor use in a computing environment. A computer program does notnecessarily correspond to a file in a file system. A program may bestored in a portion of a file that holds other programs or data (e.g.,one or more scripts stored in a markup language document), in a singlefile dedicated to the program in question, or in multiple coordinatedfiles (e.g., files that store one or more modules, sub programs, orportions of code). A computer program may be deployed to be executed onone computer or on multiple computers that are located at one site ordistributed across multiple sites and interconnected by a communicationnetwork.

The processes and logic flows described in this specification may beperformed by one or more programmable processors executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows may also be performedby, and apparatus may also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any appropriate kind of digital computer.Generally, a processor may receive instructions and data from a readonly memory or a random access memory or both. Elements of a computercan include a processor for performing instructions and one or morememory devices for storing instructions and data. Generally, a computermay also include, or be operatively coupled to receive data from ortransfer data to, or both, one or more mass storage devices for storingdata, e.g., magnetic, magneto optical disks, or optical disks. However,a computer need not have such devices. Moreover, a computer may beembedded in another device, e.g., a mobile telephone, a personal digitalassistant (PDA), a mobile audio player, a Global Positioning System(GPS) receiver, to name just a few. Computer readable media suitable forstoring computer program instructions and data include all forms ofnon-volatile memory, media and memory devices, including by way ofexample semiconductor memory devices, e.g., EPROM, EEPROM, and flashmemory devices; magnetic disks, e.g., internal hard disks or removabledisks; magneto optical disks; and CD ROM and DVD-ROM disks. Theprocessor and the memory may be supplemented by, or incorporated in,special purpose logic circuitry.

To provide for interaction with a user, implementations may be realizedon a computer having a display device, e.g., a CRT (cathode ray tube) orLCD (liquid crystal display) monitor, for displaying information to theuser and a keyboard and a pointing device, e.g., a mouse or a trackball,by which the user may provide input to the computer. Other kinds ofdevices may be used to provide for interaction with a user as well; forexample, feedback provided to the user may be any appropriate form ofsensory feedback, e.g., visual feedback, auditory feedback, or tactilefeedback; and input from the user may be received in any appropriateform, including acoustic, speech, or tactile input.

Implementations may be realized in a computing system that includes aback end component, e.g., as a data server, or that includes amiddleware component, e.g., an application server, or that includes afront end component, e.g., a client computer having a graphical userinterface or a web browser through which a user may interact with animplementation, or any appropriate combination of one or more such backend, middleware, or front end components. The components of the systemmay be interconnected by any appropriate form or medium of digital datacommunication, e.g., a communication network. Examples of communicationnetworks include a local area network (“LAN”) and a wide area network(“WAN”), e.g., the Internet.

The computing system may include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specifics, these should not beconstrued as limitations on the scope of the disclosure or of what maybe claimed, but rather as descriptions of features specific toparticular implementations. Certain features that are described in thisspecification in the context of separate implementations may also beimplemented in combination in a single implementation. Conversely,various features that are described in the context of a singleimplementation may also be implemented in multiple implementationsseparately or in any suitable sub-combination. Moreover, althoughfeatures may be described above as acting in certain combinations andeven initially claimed as such, one or more features from a claimedcombination may in some examples be excised from the combination, andthe claimed combination may be directed to a sub-combination orvariation of a sub-combination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the implementations described above should not beunderstood as requiring such separation in all implementations, and itshould be understood that the described program components and systemsmay generally be integrated together in a single software product orpackaged into multiple software products.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of the disclosure. For example, various formsof the flows shown above may be used, with steps re-ordered, added, orremoved. Accordingly, other implementations are within the scope of thefollowing claims.

The invention claimed is:
 1. A computer-implemented method performed bya platform executed by one or more computing devices, the methodcomprising: a computer server receiving an electronic transmission ofassessment data from at least one external assessment service; theassessment date comprising at least one assessment having at least onerespective question, category data describing at least one categoryassociated with each respective question of the at least one assessment,original score data describing a plurality of scores each assigned basedon a response of an individual to the at least one respective questionon the at least one assessment; the computer server normalizing theassessment data wherein the original scores are assigned a normalizedscore based on a uniform scoring scale; the computer server analyzingthe assessment data to create analysis results, said analysis resultscomprising aggregate score data for each respective question of the atleast one assessment; the computer server electronically transmittingthe analysis results to at least one interactive dashboard accessiblevia a user computing device; and the computer server receiving, throughthe at least one interactive dashboard accessible via said usercomputing device, a request to create a unique ticket in a continuousquality improvement (CQI) system that is accessible through theplatform; in response to the request, the computer server creating saidunique ticket in the CQI system, wherein the unique ticket includes ascreenshot of the data that was presented on the interactive dashboardof the user device at the time the ticket was created; wherein theunique ticket is assigned unique meta data and stored on the platform;and wherein the platform comprises an analysis engine having a pluralityof modules, the at least one interactive dashboard through which theanalysis results are presented, and a reporting interface, saidplatform, said analysis engine, said plurality of modules, said at leastone interactive dashboard, and said reporting interface comprising amedium through which the receiving, normalizing, analyzing, presentingand receiving steps occur.
 2. The method of claim 1, wherein: thecomputer server further analyzes the assessment data by determining, foreach respective question, a point biserial and a percentage correctmetric that indicates a percentage of students who answered the questioncorrectly, based on the score data for the respective question; anddetermining a quality of each question based on its point biserial andpercentage correct metric; and the computer server generates a reportidentifying the question determined to be low quality because the pointbiserial and percentage correct are below a preset threshold value. 3.The method of claim 1, further comprising: receiving, through the atleast one interactive dashboard provided by the platform, an indicationthat the ticket is to be resolved and, in response: resolving theticket; and updating the ticket to include the screenshot of a currentpresentation of the at least one interactive dashboard when the ticketis resolved.
 4. The method of claim 3, further comprising: presenting,through the at least one interaction dashboard, change information thatdescribes a different in at least a portion of the assessment data fromwhen the ticket is resolved relative to when the ticket was created. 5.The method of claim 1, wherein the request to create the ticketindicates one or more of a question, an assessment that includes thequestion, a course in which the assessment is administered, and anindividual associated with the course.
 6. The method of claim 1,wherein: the request to create the ticket indicates at least one actionto be performed, at least one responsible individual to perform the atleast one action; and a notification is sent to the at least oneresponsible individual responsive to creating the ticket in the CQIsystem.
 7. The method of claim 1, further comprising: generating,through the at least one interactive dashboard, a report that includes alist of tickets in the CQI system that have been created based onrequests submitted through the at least one interactive dashboard.
 8. Asystem comprising: at least one processor; and a memory communicativelycoupled to the processor, the memory storing instructions which, whenexecuted by the at least one processor, instruct the at least oneprocessor to perform operations comprising: receiving an electronictransmission of assessment data from at least one external assessmentservice; the assessment data comprising at least one assessment havingat least one respective question, category data describing at least onecategory associated with each respective question of the at least oneassessment, original score data describing a plurality of scores eachassigned based on a response of an individual to the at least onerespective question on the at least one assessment; normalizing theassessment data wherein the original scores are assigned a normalizedscore based on a uniform scoring scale; analyzing the assessment data tocreate analysis results, said analysis results including aggregate scoredata for each respective question of the at least one assessment;electronically transmitting the analysis results to at least oneinteractive dashboard accessible via a user computing device; receiving,through the at least one interactive dashboard accessible via said usercomputing device a request to create a unique ticket in a continuousquality improvement (CQI) system that is accessible through theplatform; in response to the request, creating said unique ticket in theCQI system, wherein the unique ticket includes a screenshot of the datathat was presented on the of the interactive dashboard of the userdevice at the time the ticket is created; wherein the unique ticket isassigned unique meta data and stored on the platform; and wherein theplatform comprises an analysis engine having a plurality of modules, theat least one interactive dashboard through which the analysis resultsare presented, and a reporting interface, said platform, said analysisengine, said plurality of modules, said at least one interactivedashboard, and said reporting interface comprising a medium throughwhich the receiving, normalizing, analyzing, presenting and receivingsteps occur.
 9. The system of claim 8, wherein: analyzing the assessmentdata further includes: determining, for each respective question, apoint biserial and a percentage correct metric that indicates apercentage of students who answered the question correctly, based on thescore data for the respective question; and determining a quality ofeach question based on its point biserial and percentage correct metric;and the request to create the ticket indicates at least one question forwhich the determined quality is low, as indicated by the point biserialand percentage correct metric each being below a respective thresholdvalue.
 10. The system of claim 8, the operations further comprising:receiving, through the at least one interactive dashboard provided bythe platform, an indication that the ticket is to be resolved and, inresponse: resolving the ticket; and updating the ticket to include thescreenshot of a current presentation of the at least one interactivedashboard when the ticket is resolved.
 11. The system of claim 10, theoperations further comprising: presenting, through the at least oneinteraction dashboard, change information that describes a different inat least a portion of the assessment data from when the ticket isresolved relative to when the ticket was created.
 12. The system ofclaim 8, wherein the request to create the ticket indicates one or moreof a question, an assessment that includes the question, a course inwhich the assessment is administered, and an individual associated withthe course.
 13. The system of claim 8, wherein: the request to createthe ticket indicates at least one action to be performed, at least oneresponsible individual to perform the at least one action; and anotification is sent to the at least one responsible individualresponsive to creating the ticket in the CQI system.
 14. The system ofclaim 8, the operations further comprising: generating, through the atleast one interactive dashboard, a report that includes a list oftickets in the CQI system that have been created based on requestssubmitted through the at least one interactive dashboard.
 15. One ormore non-transitory computer-readable storage media storing instructionswhich, when executed, cause at least one processor to perform operationscomprising: receiving an electronic transmission of assessment data fromat least one external assessment service; the assessment data comprisingat least one assessment having at least one respective question,category data describing at least one category associated with eachrespective question of the at least one assessment, score datadescribing a plurality of scores each assigned based on a response of anindividual to the at least one respective question on the at least oneassessment; normalizing, by the platform, the assessment data whereinthe original scores are assigned a normalized score based on a uniformscoring scale; analyzing the assessment data to create analysis results,said analysis results comprising aggregate score data for eachrespective question of the at least one assessment; electronicallytransmitting the analysis results to at least one interactive dashboardaccessible via a user computing device; receiving, through the at leastone interactive dashboard provided by the platform, a request to createa unique ticket in a continuous quality improvement (CQI) system that isaccessible through the platform; in response to the request, creatingsaid unique ticket in the CQI system, wherein the unique ticket includesa screenshot of the data that was presented on the interactive dashboardat the time the ticket is created; wherein the unique ticket is assignedunique meta data and stored on the platform; and wherein the platformcomprises an analysis engine having a plurality of modules, the at leastone interactive dashboard through which the analysis results arepresented, and a reporting interface, said platform, said analysisengine, said plurality of modules, said at least one interactivedashboard, and said reporting interface comprising a medium throughwhich the receiving, normalizing, analyzing, presenting and receivingsteps occur.
 16. The one or more non-transitory computer-readablestorage media of claim 15, wherein: analyzing the assessment datafurther includes: determining, for each respective question, a pointbiserial and a percentage correct metric that indicates a percentage ofstudents who answered the question correctly, based on the score datafor the respective question; and determining a quality of each questionbased on its point biserial and percentage correct metric; and therequest to create the ticket indicates at least one question for whichthe determined quality is low, as indicated by the point biserial andpercentage correct metric each being below a respective threshold value.17. The one or more non-transitory computer-readable storage media ofclaim 15, the operations further comprising: receiving, through the atleast one interactive dashboard provided by the platform, an indicationthat the ticket is to be resolved and, in response: resolving theticket; and updating the ticket to include the screenshot of a currentpresentation of the at least one interactive dashboard when the ticketis resolved.